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Foreword 
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a.  SUBPROCESS  *, 


AGGREGATION  OF  CONDITIONAL  ABSORBING  MARKOV  CHAINS 


C.  Bernard  Barfoot 


Canter  for  Naval  Analyses 
Alexandria,  Virginia  U.S.A. 


When  modeling  a  process  by  naans  of  a  finite  Markov  chain,  it  is  sonatinas  necessary 
or  desirable  to  stratify  the  process  into  subprocesses  and  nodal  each  of  these 
subprocesses.  The  resulting  Markov  chain  for  each  aubprocess  becomes  a  conditional 
Markov  chain  in  that  its  transition  probabilities  are  relative  to  its  associated 
subprocess.  This  paper  derives  the  method  for  aggregating  conditional  absorbing 
Markov  chains  (each  of  which  has  the  same  state  space)  into  a  single  (unconditional) 
chain  that  is  representative  of  the  total  process  and  haa  the  sane  state  space  as  the 
conditional  chains. 


1.  INTRODUCTION 

When  modeling  a  process  by  means  of  a  finite 
Markov  chain,  it  is  sometimes  necessary  or 
desirable  to  stratify  the  process  into 
subprocesses  and  model  each  of  these 
individual  subprocesses.  For  example,  in  a 
study  of  a  distributed  data  base  system  [1], 
the  flow  of  data  was  modeled  as  a  Markov  chain 
for  several  separata  geographic  locatlona.  In 
a  recruiting  study  [2] ,  the  movement  of 
military-age  men  through  the  recruiting 
process  and  into  the  armed  forces  was  modeled 
for  separate  racial  and  educational  groups  as 
a  Markov  chain  with  a  single  state  space  for 
each  group — only  the  input  data  (transition 
probabilities)  to  the  modal  were  changed  for 
each  group.  In  [3],  a  Markov  chain  model  was 
used  to  investigate  the  consequences  of 
Induced  abortion  for  'different  groups  of  woman 
by  estimating  transition  probabilities 
separately  for  each  group.  v 

When  the  above  procedure  is  used,  the 
resulting  Markov  chain  for  each  subprocess 
becomes  a  conditional  Markov  chain  in  that  its 
transition  probabilities  are  relative  to  its 
associated  subprocess.  This  paper  derives  the 
method  for  aggregating  these  separate 
conditional  chains  (each  of  which  has  the  same 
state  space)  into  a  single  (unconditional) 
chain  that  is  representative  of  the  total 
process  and  has  the  same  state  space  as  the 
conditional  chains. 

2.  ILLUSTRATIVE  EXAMPLE 

Figure  1  illustrates  a  simple  four-state 
process  I  chat  has  been  stratified  into  two 
subprocessas  and  Tj.  Suppose  data  have 
been  collected  for  each  subprocaas  (which 
might  represent  different  geographic  regions 
or  different  groups  of  people,  for  example) 
and  transition  probabilities  have  been 
estimated  as  shown  in  the  aatrlces  of 
transition  probabilities  P,  and  P, .  Further, 
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b  SUBPROCESS  +, 


a  PROCESS  ♦ 

FIG.  1:  FOUR-STATE  PROCESS 
WITH  TWO  SUBPROCESSES 


suppose  we  know  ths  fraction  of  ciaa  (i  .a. , 
tha  probability)  that  the  process  originates 
in  each  aubprocass,  say  f^  and  f2>  and  also 
know  that  the  process  always  begins  in  state 
S,.  Given  this  information,  how  do  we 
determine  P?  This  exanple  illustrates  the 
general  problea  addressed  in  this  paper.  As 
we  shall  see  later  in  the  paper,  what  aight  be 
considered  as  cwo  "obvious"  methods  of 
determining  P  do  not,  in  general,  work: 

(1)  aggregating  the  data  from  the  subprocesses 
and  (2)  defining  P  -  f^  +  fjPj. 

3.  DERIVATION  OF  P 

Suppose  the  process  T  under  study  can  be 
stratified  into  a  subprocesses  ?k, 

(k  ■  1,  ...,  a),  each  with  Che  same  state 
space.  We  assume  that  V  and  all  the  Tg  are 
being  modeled  as  an  irreducible,  finite, 
absorbing  Harkov  chain  having  q  tranaient 
states  and  r  absorbing  states. 

If  we  number  the  states  of  .the  chain  so  that 
the  tranaient  states  "precede"  the  absorbing 
states,  then  the  matrix  Pv  of  transition 
probabilities  for  subproceas  can  be 
partitioned  as  follows: 


where  I  is  the  identity  matrix  of  order  r,  0 
is  the  c  x  q  zero  matrix,  \  is  Che  q  x  r 
matrix  containing  the  transition  probabilities 
from  transient  to  absorbing  states,  and  la 
the  aatrlx  of  order  q  containing  the 
transition  probabilities  among  the  transient 
states. 

The  probability  that  is  absorbed  in  state 

Sj  (J  -  1 . r),  given  that  the  process 

began  in  transient  state  (i  ■  1 . q), 

is  [41 

»k  *  (bljk)  -  (I  -  V"l*k  *  W 

where  is  the  aatrlx  that  gives  the  expected 
nuaber  of  class  chat  is  in  each 


tranaient  scat*  Sj  given  that  it  began  In  each 
transient  state  S^. 

We  assume,  without  loss  of  generality,  that 
each  always  begins  in  a  particular 

transient  state,  say  Sj.  Then  the  probability 
Chat  terminates  in  Sj  is 

blk  "  ‘{®k  “  ‘{Vk. 

where  e^  is  Che  unit  column  vector  e^  -  (6^), 
6il  is  the  Kronecker  delta,  and  e'  is  the 
transpose  of  e^. 

If  we  let  ffc  be  the  probability  that  ? 
originates  in  \  ( E^f^  «  1),  then  the 
probability  that  the  process  terminates  in  Sj 
is 

bl  “  Wlk  “  ‘{W1  -  Vi¬ 
ctor  criterion  for  determining  P  Is  that  the 
limiting  probabilities  of  absorption  obtained 
froa  P  must  be  the  same  as  those  obtained  from 
Pk  and  ffc.  In  ocher  words,  we  want  to 
determine  che  stochastic  matrix  P  of  order 
q  +  r  such  that 


and  e'(I  -  Q)_lR  -  e'^a  - 
To  obtain  an  expression  for  P,  let 

-  a  diagonal  matrix  of  order  q  whose 
diagonal  elements  are  froa  the  first 
row  of  N^ 

«?k  0 

- -  ■  a  diagonal  matrix  of 

■  J  order  q  +  r. 

Then,  as  we  shall  prove  in  subsequent 
theoreas,  the  aatrlx  P  that  satisfies  our 
criterion  is 

P  -  (VkV'VkVk  •  <l> 

Also,  if  P  is  given  by  (1),  Chen  the 


-2- 


submatrices  Q  and  R  are 


Q  ->  <Vk  Nfk)  1  VltHlkQR 
and  R  -  (Vk  "ik^WlkV 
4.  VERIFICATION  OF  P 

To  show  that  P  la  In  fact  the  daalred  matrix 
we  need  to  show  that: 

(1)  P  Is  stochastic;  l.e.,  p1j  >  0,  all  1 
and  j,  and  Ejp^j  “  l,  all  1. 

(2)  .{(I  -  Q)-lR  -  «{Vk«  -  V~V 

We  show  that  these  conditions  are  met  in  the 
following  two  theorems. 

Theorem  1.  P  la  stochastic. 

Proof .  From  equation  (1)  It  follows  that 
>0,  all  1  and  J,  since  each  term  Is 
nonnegatlve. 

To  show  thst  IjPjj  »  1,  all  1,  we  need  to  show 
that  Pe  -  e.  where  e  Is  the  q+r  column 
vector  all  of  whose  elements  are  unity. 

Writing  P  and  e  in  partitioned  form. 


Q  I  * 

e 

-Q.q  +  r.; 

on 

•t 

• 

r 

. 

-  - 

s  « 

How 

Q.q  +  Her  -  (Vk*fk>"J  Vk*fk«k%  +  Vr> 

”  (VkNlk)  l£kfkHlkeq  “  V 

Before  proving  the  next  theorem  we  shall  need 
the  following 

hsb*‘  «r[Vk«?k»ki  "l  ■  •  • 

Proof.  VkNlk»k“l  ■  Vk<«'«lk>»k'1 
■  Wlk»k'1 


■  Vk'i  ■ «{  •  • 

Hence  e'  -  *{[£ltflcf,lkI,k  *] 

The  existence  of  the  Inverse  will  be  shown  In 
the  next  theorem. 

Theorem  2.  e'(I  -  Q)_lR  -  e{Vk(I  ' 

Proof .  Consider  the  first  two  factors  of  the 
left  hand  side: 

«{<i  -  <»~l  -  -  (Vki,ik>'l£k£ksik<3k]"1 

*  ‘{[Vk^lk  “  £kfkNlkQk]  l£k.fkt,lk 
■  ‘{[Witc*1  -  V]"lWik 

-  •{[Vk>ik<V1>]’lEkfk!,ik 

«  e’’skfkslk  by  the  Lemma. 

Hence  the  Left  hand  side  becomes 

e'U  -  Q)-lR  -  e'(Vk<><Wlk>'1  Wlk\ 

-  WeXk>«k  • 

Now  e'N^  •  Nik,  the  first  row  of 
therefore 

*{(i  -  Q>'1*  ■  WiA  ■  VkKV«k 

■  «i>JvWk  ’  •i'Jkfk<1  - 

which  completes  the  proof. 

It  Is  now  seen  thst  the  matrix  2kfkN^kNk-1' 
considered  In  the  Lemma  is  nonsingular  since 
It  Is  the  product  of  two  nonslngular  matrices: 

EkfkK?kNk  1  " 

-  <  Wik>'lWiA3  • 

5.  EXPECTED  TIME  TO  ABSORPTION 

Another  quantity  of  Interest  Is  the  expected 
time  to  absorption  In  a  given  absorbing  state 


Hence  Pe 


3- 


Sj,  given  Chet  ?k  began  in  transient  state 
S^.  The  matrix  of  transition  probabilities 
for  fk  conditioned  on  the  hypothesis  that  ?k 
is  absorbed  in  Sj  Is  [4] 


Q*r'(1  -  v 


where  Qjk  - 


and  D 

jk 


is  a  diagonal  matrix  of  order  q  formed  from 
column  j  of  P  Is  of  order  q  +  1, 

e'(I  -  Qjk)  is  a  q-cooponent  column  vector, 
and  0  is  the  q-component  zero  row  vector. 

Given  chat  fk  is  absorbed  in  state  Sj,  the 
expected  time  to  absorption  is  [1] 


VJk  -  (I  -  QjkrlTk  -  NjkTk 

where  Tk  •  (tlk)  is  a  column  vector  whose  • 
elements  are  Che  expected  times  clk  that  ?k 
spends  in  each  transient  state  . 

Since  fk  always  begins  in  transient  state  S|, 
the  expected  time  for  ?k  to  be  absorbed  in 
state  Sj  is 

«{V  - 

and  Che  expecced  time  for  the  overall  process 
f  to  be  absorbed  in  state  Sj  is 

“l7!  ”  Ckfk,l7jk  "  *lSk£k(I  "  Qjk5  ^k* 


To  determine  the  matrix  Pj  and  the  time  vector 
T  for  f  under  the  hypothesis  chat  f  is 
absorbed  in  S j,  we  use  the  criterion  that  the 
expected  time  to  absorption  obtained  by  using 
Pj  and  T  must  be  the  same  as  the  time  obtained 
by  using  Pjk,  Tk>  and  fk>  In  other  words,  we 
want  to  determine  P ,  and  T  such  chat 


and  .{(I  -  Qj)  T  -  e'Wl  -  QJk)  ‘Tk. 


To  obtain  expressions  for  Pj  and  T,  let 

Nljk  *  the  diagonal  matrix  of  order  q 

whose  diagonal  elements  are  from 
the  first  row  of  Njk, 


PJ  “  (2k£kAJk5  Vk\jk*jk  • 

QJ  *  ^  JV£ktll jk^  lzkfk‘',ljkQJk  • 


and  T  -  (Vk»ljk>“lVkHljkTk 


The  proofs  that  Pj  is  stochastic  and  that 
e'(I  -  Qj)-lT  -  e'Vtd  - 


are  the  same  as  those  given  la  Theorems  1 
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